Deep Learning Services (DLS) when utilizing confidential or otherwise sensitive information often offer a data owner to host an in-house or private DLS in order to ensure data confidentiality. An in-house DLS may not be viable option in all instances and are often expensive solutions. In order to safeguard confidential or otherwise sensitive information the data owner may utilize an Encrypted Deep Learning Service (EDLS) hosted by an EDLS provider. However, to configure the EDLS the data owner is often forced to reveal the confidential or sensitive information. An EDLS which accepts encrypted data for training and inference could minimize or eliminate the risk of the confidential or sensitive information being revealed to unauthorized parties.